Individual-level variation in reference game data

Michael Franke & Judith Degen

of monsters and robots

simple & complex reference games

 

RSAstim

Franke & Degen (2016), Reasoning in reference games, PLoS one 11(5)

complexity of reasoning

chains

individual variation in reasoning depth

typePredictions

   

formulas

(c.f., Camerer 2006, Franke 2011, Jaeger 2014)

data

participants

  • 60 subjects each for production & comprehension
    • 12 trials of each critical condition per subject
  • 129 subjects in salience prior elicitation experiment

production data

   

restults1

comprehension data

   

results2

modeling

population-level modeling

  • maximum-likelihood fit of RSA model to population-level data
  • correlation \(r = 0.997\), \(p < 0.0001\)

 

restults1

data-generating model

modelgraph

posterior over types \(P(\tau_i \mid D)\)

bysubject

model comparison: general

\[\underbrace{\frac{P(M_1 \mid D)}{P(M_2 \mid D)}}_{\text{posterior odds}} = \underbrace{\frac{P(D \mid M_1)}{P(D \mid M_2)}}_{\text{Bayes factor}} \ \underbrace{\frac{P(M_1)}{P(M_2)}}_{\text{prior odds}}\]

  • Bayes factors give the factor by which prior odds should be adjusted in light of the data
  • Bayes factors of above 10 (or below 0.1) are conventionally considered noteworthy evidence in favor of one model over the other
  • Bayes factors compare the likelihood of the data under the prior predictive distribution, since:

\[ P(D \mid M_i) = \int P(\theta \mid M_i) \ P(D \mid \theta, M_i) \ \text{d}\theta \] - Bayes factors thereby implicitly penalize models with wide-spread a priori predictions and favor models which make precise predictions (if these are borne out by the data) - as models with

model comparison: applied

we use the Savage-Dickey method to approximate Bayes factors:

calculations

  • visual impression corroborated by Bayesian model comparison:
    • RSA’s assumption that speaker population is homogeneously of level-1 is tenable
    • RSA’s assumption that speaker population is homogeneously of level-1 is not tenable

fini

upshot

 

  • individual-differences should be taken into account
    • in line with mixed effects factors in regression

 

  • (Bayesian) mixture models are intuitive tools to formalize theories about individual differences
    • they can go beyond mixed effects factors in regression!

 

  • the data should decide how “rational” and “interactive” each subject is

 

  • probabilistic pragmatics blends into tailor-made statistical modeling